System and method for creating and implementing scalable and effective multi-media objects with human interaction proof (hip) capabilities

ABSTRACT

Embodiments of the invention provide a method and apparatus (“system”) that overcome the above-mentioned problems among others and provide an innovative solution aimed at creating an interactive, dynamic and effective multi-media object with HIP capabilities which may be used in online advertising, security, and user-defined security. The system leverages the existing HIP CAPTCHA real estate to create multi-media objects that guarantee a captivated audience, especially in online advertising. Combining interactive multi-media objects with HIP capabilities helps to meet a very critical need faced by advertisers and websites today—creating an effective impression of any multi-media object on a user (a guaranteed eyeball). Embodiments of the current invention introduce a variety of formats that involve interacting with a multi-media object to provide a more natural user interaction and ease of use while maintaining security.

CROSS REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of U.S. patent applicationSer. No. 13/975,084, CREATING AND IMPLEMENTING SCALABLE AND EFFECTIVEMULTIMEDIA OBJECTS WITH HUMAN INTERACTION PROOF (HIP) CAPABILITIES, WITHCHALLENGES COMPRISING SECRET QUESTION AND ANSWER CREATED BY USER, ANDADVERTISEMENT CORRESPONDING TO THE SECRET QUESTION, filed Aug. 23, 2013,which is a continuation-in-part of U.S. patent application Ser. No.:13/566,690, SYSTEM AND METHOD FOR CREATING AND IMPLEMENTING DYNAMIC,INTERACTIVE AND EFFECTIVE MULTI-MEDIA OBJECTS WITH HUMAN INTERACTIONPROOF (HIP) CAPABILITIES, filed Aug. 3, 2012, which claims benefit ofU.S. Provisional Patent Application Ser. No.: 61/515,636, SYSTEM ANDMETHOD FOR CREATING AND IMPLEMENTING DYNAMIC, INTERACTIVE AND EFFECTIVEMULTI-MEDIA OBJECTS WITH HUMAN INTERACTION PROOF (HIP) CAPABILITIES,filed Aug. 5, 2011, and the 13/975,084 patent application also claimsbenefit of U.S. Provisional Patent Application Ser. No.: 61/692,343,SYSTEM AND METHOD FOR CREATING AND IMPLEMENTING SCALABLE AND EFFECTIVEMULTI-MEDIA OBJECTS WITH HUMAN INTERACTION PROOF (HIP) CAPABILITIES,filed Aug. 23, 2012, the entirety of each of which is incorporatedherein by this reference thereto.

BACKGROUND OF THE INVENTION

Technical Field

This invention relates generally to the field of multimedia objects andhuman interactive proof (HIP) technology. More specifically, techniquesare provided that involve interacting with a multimedia object toprovide a more natural user interaction and ease of use whilemaintaining security.

Description Of The Related Art

The growth of internet usage across different interfaces and devices hasled to widespread adoption across all demographics. The number ofwebsites is ever increasing and more financial transactions happen onthe internet. Websites are widely used to provide users with aconvenient means to order products such as tickets, to access personalaccount information, to open new email accounts, to add content toexisting content like comments on blogs, uploading multi-media contenton websites or to access other services. These systems are not onlyconvenient to website owners as well as to their users, but they alsoreduce overall costs.

Unfortunately such systems can also provide a loophole, through whichhackers can obtain access to personal or other restricted data, disruptservices, poison existing content with irrelevant information decliningthe value of existing content, consume all the resources of the websitefor malicious activities and distribute worms or spam. This is donecommonly through the use of automated bots, scripts, and other maliciouscomputer programs.

This has led to the development of Human Interactive Proof (HIP)technology. HIPs are challenges designed to be solved by humans, so thatthey aren't discouraged from using a service. At the same time the HIPmust be difficult enough to make the cost of developing or processing amalicious bot, script to break it, uneconomical and challenging. Oneform of such HIP which is widely used is CAPTCHAs (“Completely AutomatedPublic Turing test to tell Computers and Humans Apart”, also referred toherein as “captcha” or “captchas,” i.e. not in full caps). Captchatechnology is a challenge-response system as an attempt to ensure theresponse is from a human and this helps to isolate the human trafficfrom the malicious bot, script traffic. Captchas are fast becomingpervasive across the internet as sites move towards collaborativeuser-generated content making the need for distinguishing genuine usersfrom bots and scripts, and maintaining user privacy more critical.However, captchas may not be very user friendly and occupy precious realestate without providing any monetization.

Online advertising has emerged as an important element of anyadvertising campaign. However, the effectiveness of the existing adformats such as online banner ads is far from optimal. Several bannerads on a webpage crave for attention and has led to a phenomenon calledbanner blindness. Banner blindness is a phenomenon in web usabilitywhere visitors on a website ignore banner-like information. And webusability tests are regularly proving that knowingly or unknowingly theusers are ignoring banner ads or images which appear in banner form.

SUMMARY OF THE INVENTION

Embodiments of the invention provide a method and apparatus (“system”)that overcome the above-mentioned problems among others and provide aninnovative solution aimed at creating an interactive, dynamic andeffective multi-media object with HIP capabilities which may be used inonline advertising, security, and user-defined security. The systemleverages the existing HIP CAPTCHA real estate to create multi-mediaobjects that guarantee a captivated audience, especially in onlineadvertising. Combining interactive multi-media objects with HIPcapabilities helps to meet a very critical need faced by advertisers andwebsites today—creating an effective impression of any multi-mediaobject on a user (a guaranteed eyeball). Embodiments of the currentinvention introduce a variety of formats that involve interacting with amulti-media object to provide a more natural user interaction and easeof use while maintaining security.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an architectural diagram of a system for generating andserving scalable and effective multimedia objects with human interactionproof (HIP) capabilities on a webpage or an application, according to anembodiment;

FIG. 2 is a flow diagram of a video based HIP being created andassociated to a user profile, according to an embodiment;

FIG. 3 is a flow diagram of a user specified video and its use as achallenge to identify the correct user, according to an embodiment;

FIG. 4 is a flow diagram and an example of stitching user specifiedvideo and user defined multiple images and using the resultantmultimedia as a challenge to identify the correct user, according to anembodiment;

FIG. 5 is a schematic diagram of two secret questions and theirrespective MMO with HIP enabled, according to an embodiment;

FIG. 6 is a sample webpage or application demonstrating interactivemultimedia (MM)) serving HIP features, in which the valid response forthis is a combination from the image and response dependent on thecontent where the MMO is being served, according to an embodiment;

FIG. 7 is a schematic diagram of a webpage or application that uses webmapping technology to act as an interactive MMO that helps identify theuser as well as serve the purpose of HIP, according to an embodiment;

FIG. 7A is a schematic diagram of a registration webpage or applicationthat uses web mapping technology to act as an interactive MMO that helpsidentify the user as well as serve the purpose of HIP, according to anembodiment;

FIG. 8 is a flow diagram of creating challenges that can scale for amedia with HIP support, according to an embodiment;

FIG. 9 is a flow diagram of a method for creating hybrid challenges andresponses, according to an embodiment;

FIG. 9A is a schematic diagram of examples that demonstrate hybridchallenge and response based HIP, according to an embodiment;

FIG. 10 is a schematic diagram of noise elements being added to increasethe difficulty for bots to identify the valid objects, according to anembodiment;

FIG. 11 is a flow diagram of creating a template based HIP, according toan embodiment;

FIG. 12 is a schematic diagram of an example challenge to demonstratethe template based HIP and examples of relationships, objects,interactions, and templates, according to an embodiment;

FIG. 13 is a flow diagram of creating an HIP system which can integratesome human computation methods to utilize human interactions to solvesome computational problems, according to an embodiment;

FIG. 13A is a schematic diagram of one example to find image outliers inone image sets, according to an embodiment; and

FIG. 14 is a block schematic diagram of a system in the exemplary formof a computer system, according to an embodiment.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the invention provide a method and apparatus (“system”)that overcome the above-mentioned problems among others and provide aninnovative solution aimed at creating an interactive, dynamic andeffective multi-media object with HIP capabilities which may be used inonline advertising, security, and user-defined security. The systemleverages the existing HIP CAPTCHA real estate to create multi-mediaobjects that guarantee a captivated audience, especially in onlineadvertising. The idea of combining interactive multi-media objects withHIP capabilities helps to meet a very critical need faced by advertisersand websites today—creating an effective impression of any multi-mediaobject on a user (a guaranteed eyeball). Also, the current format ofCAPTCHAs involve deciphering distorted text which is usually difficultto do and causes a break in the natural flow of the user's interactionwith the web site. Embodiments of the current invention, aims tointroduce a variety of formats that involve interacting with amulti-media object to provide a more natural user interaction and easeof use while maintaining security.

Terminology

For purposes of understanding embodiments herein, the followingterminology and acronyms may be used:

A Multimedia Object (MMO) may be any multimedia item such as an onlineadvertisement, a static image, a video, an audio, and text. Further, itshould be appreciated that such MMO may or may not have HIP embedded init. An MMO may be interactive or non-interactive. An MMO may becomprised of rich media for example.

Multimedia object server (MMOS) is a server which serves MMOs asdescribed hereinabove.

Validation and Verification server (VS) is a server which validates andverifies user response against the HIP challenge.

Multimedia object administration server (MMOAS) is a server whichadministrates the MMOs and contains additional information, e.g.metadata, for all MMOs.

HIP—Human Interaction Proof is a technology that is presented via aprogrammed system to a user to help ensure that a human being, asopposed to an automated system, is interacting with the programmedsystem. It is generally presented as a challenge to the user, and isalso known as a CAPTCHA or a HIP challenge or a HIP CAPTCHA, for which avalid response from user is expected. A challenge of HIP can bedescribed as any system interaction which system generates and sends tothe user to make and enable user to perform the intended interaction onthe system to ensure that the interaction is made only from a human. Aresponse can be described as the interactions with the system which aredone on the posed challenge to ensure that the user who interacted ishuman. For example, the challenge can be a text message “drag thispicture from point A to point B” appearing on computer screen, for whichthe response is the dragging action of the picture by the user on thecomputer screen. It should be appreciated that a Human Interaction Proofsystem may or may not include multimedia objects.

Analytics Server (AS) is a server which performs operations includingstoring all the session information in a database configured to reportand provide analytics.

Video Image Integration Server (VIIS) is a server which combines inputvideo with other multi-media, e.g. images, and generates a resultantvideo, also referred to as a combined video, with or without metadata,e.g. embedded tags.

Status Server Module (SUS) is a module or application that stores howmany successes or failures happened for a session to allow a limitednumber of tries to identify the user rightly, before locking the user idfor too many failed attempts.

Summary of Features

Following is a summary list of features that are provided by one or moreembodiments herein. Detailed discussion about each feature is providedhereinbelow.

Combination of user specified video and used as a challenge to identifythe right user.

Combination of user specified video+multiple images which are userdefined and used as a challenge to identify the right user.

Secret questions used to validate a user's identity are classified withcertain ad types and every time the secret questions are shown,corresponding ads are shown and used as a challenge.

Interactive Multimedia Object (MMO) serving HIP (Human InteractionProof) features and whose valid responses are a combination of responsefrom the image and response dependent on the content where this MMO isbeing served. This is a multi-phase validation using the content as oneor multiple sources and the interaction as another source.

Use web mapping technology to act as an interactive MMO which helpsidentify the user as well as serve the purpose of HIP. A high leveldataflow diagram is illustrated in FIG. 1.

Methods to create a scalable challenge for a media to provide HIPsupport.

Methods to create hybrid challenges and response system to increase theefficiency of HIP.

Methods to measure engagement on HIP media.

Methods to provide multiple known images with some unknown images thatact as noise.

Methods to translate languages using HIP media.

Methods to use contextual and demography based international languagesto create challenge and response systems.

Methods to provide support for in-premise and house-media topublishers/advertisers/media platforms.

Multi-factor authentication agent.

Methods to provide flexibility to pick media source.

Methods to provide HIP support to Smart TV.

Text based ad HIP.

Template based HIP.

Human computation methods using the system.

Social categorization using HIP.

HIP Analytics.

Tools with HIP enablement feature.

It should be appreciated that the above-cited list of features is notmeant to be exhaustive and are for illustrative purposes forunderstanding embodiments herein.

DATA FLOW BETWEEN KEY COMPONENTS—AN EXEMPLARY EMBODIMENT

An embodiment can be understood with reference to FIG. 1, anarchitectural diagram of a system for generating and serving scalableand effective multimedia objects with human interaction proof (HIP)capabilities on a webpage or an application. A user opens a webpage orapplication 102 from any device (not shown), such as but not limited to,mobile, tablet, computer, internet protocol television (IPTV) console,or game console (Step 1 a). Webpage or application 102 comprises awidget (not shown) which is configured to request an MMO and a challengefrom an MMOS 106 (Step 1 b), which is a component of an engagementserver 104. A description of an embodiment of the widget can be found inco-assigned patent application Ser. No. 13/566,690, which isincorporated herein by this reference thereto. The user is presentedwith the MMO which has a challenge. For illustrative purposes only, thewebpage or application shows a video and an accompanying question,“Click on your mother-in-law in the video.” After the user responds tothe challenge, the response is sent from webpage or application 102 to avalidation server 108 of engagement server 104 (Step 2). Validationserver 108 determines whether the response matches a stored, validatingresponse and sends an a success or failure message back to webpage orapplication 102 (Step 3). When the message is the success message,webpage or application 102 sends an acknowledgement back to a challengeand response repository 110 of engagement server 104 (Step 4). When themessage is the failure message, webpage or application 102 sends a newrequest to engagement server 104, which specifically is received by MMOS106, until a predetermined threshold number of times of sending a newrequest upon failure is met (Step 4).

In an embodiment, engagement server 104 also comprises a VIIS 112. VIIS112 receives a request from MMOS 106 for an MMO containing a video basedon received filter criteria. Examples of such filter criteria aredescribed hereinbelow regarding MMOs having or being video. VIIS 112finds a matching MMO with video based on the filter criteria or profileinformation and returns such MMO to MMOS 106.

In an embodiment, engagement server 104 also comprises an AS 114. AS 114receives any details and metadata regarding the request from MMOS 106and any details and metadata regarding the response from VS 108. AS 114is configured to store such informational data, perform analytics onsuch informational data, and provide results, such as in report format,on such analyzed and raw informational data. An exemplary AS 114 isdescribed in co-assigned patent application Ser. No. 13/566,690, whichis incorporated herein by this reference thereto.

In an embodiment, engagement server 104 also comprises a SUS 116. SUS116 receives and records the occurrences of successful and failedresponses from VS 108. The communication is bi-directional should VS 108need to access data in SUS 116.

In an embodiment, VS 108 requests challenge and response data fromchallenge and response repository 110. The communication isbi-directional and challenge and response repository 110 may alsorequest data from VS 108.

A USER SPECIFIED VIDEO AND ITS USE AS A CHALLENGE TO IDENTIFY THE RIGHTUSER

Typically, the MMOs used to isolate malicious bots on websites havevarious modes of security to distinguish human traffic, known as HIP.Standard HIP proof is implemented at the website level. However,embodiments herein address the user level and uses the same MMO toidentify a specific user and to distinguish HIP from bots. This userspecific security HIP module (UHIP) which acts both as a HIP proof andalso as a security mechanism may be implemented in the cloud or in anappliance. For instance, a user defined video may be chosen to uniquelyidentify the user and also as the HIP proof, the details of which arediscussed herein.

For example, a user uploads his/her marriage video and tags people andinteresting details about the people. This is used by a randomizer tochallenge the user later on to prove his identity. In an embodiment, therandomizer is a module which is part of MMOS and picks a random video idand question id for a specific user and such information is fetched fromthe database/storage repository at a later point in time. For instance,the user may be asked to please click on the brother's father-in-law orto click on the bride's grandmother and to type in her name.

An embodiment is described below.

1. A user goes to a website which needs to identify users uniquely. Alsothe website uses HIP to filter out non-human traffic.

2. The user is presented with an option to upload a video of his choice.The video may be from various sources including but not limited to localcomputer, cloud, private repository, public domain or network. On thevideo, using any standard video editing software, which is applicationsoftware that handles post-production video editing of digital videosequences on a computer, the user may tag pertinent information at theframe level. The embodiment creates image maps at the frame level oftags of objects in the frame.

3. The video along with the tags are associated to the user's profileand stored persistently in a database.

One or more embodiments can be understood with reference to FIG. 2, asample flow of a video based HIP is created and associated to a userprofile, and as described hereinbelow.

Step 201) In a User signup/registration form, the user profile detailssuch as but not limited to UserlD, password, address, etc. are asked. Aswell, an option is provided to upload video, or other multimedia contentin other use cases, and to create secret questions and answers. Sessioninformation from the page is sent to MMOS to provide a unique id for theuser.

Step 202) A video file is provided and the file may be located locallyor at any http URL. Along with such file additional images may also beprovided. The system creates unique ids for the video and any images andassociates such unique ids to the unique id of the user. The video fileand the images information is sent to MMOS and the unique ids for videoand images are created and assigned to the user id from the previousstep.

Step 203) With use of video editing software, the individual frames orsections of the video are extracted and shown as a list to the user. Theextracted frames or sections are assigned unique ids and associated tothe unique video id, from the previous step. The unique ids for theframes and sections are generated from MMOS after information from thisstep is sent. These ids are tied to the video id from the previous step.

Step 204) The user chooses a specific section or frame and highlights aspecific area or object in the image and tags it with a question and ananswer for that question. For example, in the figure in a particularframe, the user may highlight the lady and specify “Type the name ofyour mother” and the corresponding answer would be “Anne Paul”. Anothervariation of the question for smartphones may be “tap on your mother'sface” and the co-ordinates from the highlighted section are stored as ananswer. As well, one or many question and answer sets may be created ona specific frame/section of the video or across multiple frames/sectionsof the video.

Step 205) The data from step 204) are submitted to MMOS. Unique questionid and answer id are generated by MMOS and associated with the frame idof the video id from previous steps. All such ids are tied to the userid.

Step 206) The data from step 205) are stored in the database/storagerepository.

The steps described hereinabove explain one or more embodiments of thevideo tagging process.

Following are further steps describing one or more embodiments for theserving process for user identification (not pictured.)

Next, when the user identification is required and when there is arequirement to filter out no-human traffic (HIP), the user is posed withthe video and asked by the randomizer module to identify a randomlychosen image from the video, which has earlier been tagged. Therandomizer module is part of MMOS and picks a random video id andquestion id for the specific user. Such information is fetched from thedatabase/storage repository. MMOS sends the video and the metadatarelated to the question and video to the requesting page. It also sendsin device information requesting the information such as but not limitedto desktop browser, smartphone, tablet, smart-tv, and so on.

The requesting webpage or application displays the video as thechallenge in an HIP form (e.g. type-in, video, audio) and has aplaceholder attribute to capture the user response. Examples may includebut are not limited to the text entered as a response, the attributes ofa click, the co-ordinates of the drag and drop action on the MMO, theresponse from haptic technology on supporting devices, and touch andtactile sensors on supporting devices. From the example described inFIG. 2, for the specific user, the requesting webpage or applicationmight load the video and one of the questions may be “Type the name ofyour mother”.

MMOS also parses the requesting device information and has a preset listof device capabilities in the database such as but not limited to tap,swipe, gesture, and audio capabilities of the device. MMOS uses thedevice information to choose the HIP, e.g. questions (see the FIG. 2example) such as “tap on your mother's face,” based on the devicecapabilities. It should be appreciated that such capability of MMOS isnot limited to the discussion of and use of video, but applies torequesting any MMOS with HIP capability from any type of device.

A session id is created by MMOS and information about the request suchas time requested, user id, video id, question id, and device type arestored in the database about the session sent to the requesting page.

The user interacts with the video and the interaction attributes, e.g.the text entered as a response, the attributes of a click, theco-ordinates of the drag and drop action on the MMO, the haptictechnology on supporting devices, and touch and tactile sensors onsupporting devices such as touch-screen monitors, computers,smart-phones, and game consoles, are sent to the validation server (VS).

The Validation Server (VS) validates the response attributes to thechallenge served for the MMO and verifies it with the dataset for apossible match. VS verifies the response sent for the session id andcompares it to the data stored in the database for validity of theresponse for the question sent for that frame id, video id, and user idfor that session.

When the response satisfies the challenge, a success message is sent tothe webpage by the MMOS and statistics about the request, e.g. thetimestamp, website which requested, user details if exists, and IPaddress, are sent to the Analytics Server (AS). AS stores the sessioninformation in a database intended for reporting and providinganalytics.

When the response does not fulfill the challenge posed, VS sends afailure notification to the webpage and updates the Status Server Module(SUS) with the failed status. SUS stores how many successes or failuresoccurred for a session and allows a predetermined, limited number oftries to identify the user correctly, before locking the user id for toomany failed attempts. This is to prevent brutal attack mechanisms tobreaking into a user's account. The threshold of tolerated failedattempts is stored in a database associated with SUS.

The webpage indicates failure to the user and sends a request for a newMMO with a challenge to the MMOS, when the threshold for allowablefailures is still unreached.

The process repeats from the first bulleted step hereinabove unless thenumber of failures threshold is reached by the website or success ismet.

An embodiment of using a user specified video and its use as a challengeto identify the correct user can be understood with reference to FIG. 3,a flow diagram. At step 301, a user creates a profile on a webpage orapplication. The webpage or application has an GUI element for the userto choose to upload a video. At step 302, the user uploads the video. Aswell, the webpage or application is configured to allow the user toselect video and in other embodiments other content from localrepositories, the public domain, or private repositories. At step 303,the user, using video editing software, is able to create tags at theframe level for the uploaded video. The system is configured to allowthe user to creates tags at different frames within the same video. Atstep 304, the video is stored in a video repository and the tags andrelated metadata of video frames and associated tags are stored in ametadata repository. It should be appreciated that the video andmetadata may be stored in a single repository or in other embodiments,such as the cloud and that the embodiment shown in FIG. 3 is by way ofexample and should not be limiting.

STITCHING USER SPECIFIED VIDEO AND USER DEFINED MULTIPLE IMAGES ANDUSING THE RESULTANT MULTIMEDIA AS A CHALLENGE TO IDENTIFY THE CORRECTUSER

In an embodiment, a user specific security HIP module (UHIP) acts bothas HIP proof and also as a security mechanism and may be implemented onthe cloud or in an appliance. As well, in this case a user defined videois chosen to uniquely identify the user and the user has an option toinclude one or more images along with the video. The images are stitchedin the video and the resultant video is used to as a HIP mechanism. Oneskilled in the art would readily recognize that image stitching in thevideo can be performed by using various applications in the market andthat image stitching processing does not need to be taught herein.

For example, a user uploads his vacation trip video and tags many peopleand interesting details about the people. As well, he uploads severalimages of his friends. The Video Image Integration Server (VIIS) maycombine these two MMO types and generate a combined video. Such combinedvideo may be used by the randomizer module to challenge him at a latertime to prove his identity. For example, such challenge may be pleaseclick on your friend who is standing in the line to take the rollercoaster ride.

An embodiment of using the image stitching processing is describedhereinbelow.

1. A user goes to a website which needs to identify users uniquely. Alsothe website uses HIP to filter out the non-human traffic.

2. The user is presented with an option to upload a video of his choice.On the video he may tag pertinent information at the frame level.

3. Also, he may have the option to choose or upload a single or multipleimages. And for all these images he may define and associate userdefined tags persistently in a database.

4. The video and the images, along with the tags are associated to theuser's profile and stored persistently.

5. The Video Image Integration Server (VIIS) combines this video withthe images and generates a resultant video associated with this userwhich has the images and tags embedded. The combined video and createdmetadata are stored in the database/storage repository.

A sample flow in accordance with an embodiment may be understood withreference to FIG. 2, as further described hereinbelow:

Step 201) In a User signup/registration form, the user profile detailssuch as but not limited to UserlD, password, address, etc. are asked. Aswell, an option is provided to upload video, or other multimedia contentin other use cases, and to create secret questions and answers. Sessioninformation from the page is sent to MMOS to provide a unique id for theuser.

Step 202) A video file is provided and the file may be located locallyor at any http URL. Along with such file additional images may also beprovided. The system creates unique ids for the video and any images andassociates such unique ids to the unique id of the user. The video fileand the images information is sent to MMOS and the unique ids for videoand images are created and assigned to the user id from the previousstep.

Step 203) With use of video editing software, the individual frames orsections of the video are extracted and shown as a list to the user. Theextracted frames or sections are assigned unique ids and associated tothe unique video id, from the previous step. The unique ids for theframes and sections are generated from MMOS after information from thisstep is sent. These ids are tied to the video id from the previous step.The user is presented an option to insert the individual images uploadedin the previous step between different sections in the video. In astandard video editing software, a user may have options to split avideo at different sections and add images and stich such sections andimages together as one resultant video. Such capability is handled byVIIS for that session, user id, video id, frame id, and image id. Theinformation is sent from the page to MMOS and to VIIS. After conversioninto the resultant video, the information is sent back by MMOS to thepage where questions and responses need to be created.

Step 204) The user chooses a specific section or frame and highlights aspecific area or object in the image and tags it with a question and ananswer for that question. For example, in the figure in a particularframe, the user may highlight the lady and specify “Type the name ofyour best friend shown after the end of the roller coaster ride” and thecorresponding answer may be “Anne Paul”. Another variation of thequestion for smartphones may be “tap on Anne Paul” and the co-ordinatesfrom the highlighted section are stored as an answer. As well, one ormany question and answer sets may be created on a specific frame/sectionof the video or across multiple frames/sections of the video.

Step 205) The data from step 4) are submitted to MMOS. Unique questionid an d answer id are generated by MMOS and associated with the frame idof the video id from previous steps. All such ids are tied to the userid.

Step 206) The data from step 5) are stored in the database/storagerepository.

The steps described hereinabove explain one or more embodiments of thevideo tagging process.

Following are further steps describing one or more embodiments for theserving process for user identification (not pictured.)

Next, when the user identification is required and also when there is arequirement to filter out no-human traffic (HIP), the user is posed withthe combined video and asked by the randomizer module to identify arandomly chosen image from the video, which has earlier been tagged.

The webpage displays the video as the challenge in a HIP form (e.g.type-in, video, and audio) and has a placeholder attribute to capturethe user response. Examples of user responses include but are notlimited to the text entered as a response, the attributes of a click,the co-ordinates of the drag and drop action on the MMO, the responsefrom haptic technology on supporting devices, and touch and tactilesensors on supporting devices.

The user interacts with the video and the interaction attributes aresent to the validation server (VS). Examples of such attributes includebut are not limited to the text entered as a response, the attributes ofa click, the co-ordinates of the drag and drop action on the MMO, thehaptic technology on supporting devices, and touch and tactile sensorson supporting devices such as but not limited to touch-screen monitors,computers, smart-phones, game consoles.

The Validation Server (VS) validates the response attributes to thechallenge served for the MMO and verifies with the dataset for apossible match.

When the response satisfies with the challenge, a success message issent to the webpage by the MMOS and statistics about the request aresent to the AS. Examples of statistics about the request include but arenot limited to the timestamp, website which requested, user details whenthey exist, and IP address.

When the response does not fulfill the challenge posed, VS sends afailure message to the webpage and updates the SUS with the failedstatus.

The webpage indicates failure to the user and sends a request for a newMMO with a challenge to the MMOS, when the threshold for allowablefailures is still unreached.

The process repeats from the first bulleted step hereinabove unless thenumber of failures threshold is reached by the website or success ismet.

A flow of the process of stitching user specified video and user definedmultiple images and using the resultant multimedia as a challenge toidentify the correct user is illustrated in FIG. 4 and an example isillustrated in FIG. 4. The system receives a video file (401), forexample chosen by a user. The system splits the video into randomlysequenced pieces (402). The system takes user defined images, forexample, chosen by the user (403). The system determines associatedchallenges and responses, e.g. type-in and clickable hotspots, forexample as described in co-assigned patent application Ser. No.13/566,690, which is incorporated herein by this reference thereto(404). The system stitches the images, e.g. pet pictures, with the videosplits or pieces and creates a new video file (405). Then the systemassociates this new video file to a profile, e.g. a profile for thecorresponding website, a profile for the corresponding user, etc., anduses this new video file as a security challenge (406). Thus, on theillustrated webpage or application 408, the goal is for the user toprove his has access to log onto the site, e.g. a statement ispresented, “Prove you are Manoj” 410. Webpage or application 408 alsopresents a video which has been previously stitched 412. An examplechallenge, which is associated with the user, is shown, “Click on theimage when shown your pet” 414.

SECRET QUESTIONS USED TO VALIDATE A USER'S IDENTITY ARE CLASSIFIED WITHCERTAIN AD TYPES; EVERY TIME THE SECRET QUESTIONS ARE SHOWN,CORRESPONDING ADS ARE ALSO SHOWN AND USED AS A CHALLENGE

In an embodiment, secret questions and answers are shown to identify theuser uniquely. For example, the secret questions may be “what is themodel of your first car?” or “what is your pet's name?” or “which citywere you born in?”. An embodiment associates these questions withcertain ad types. Every time there is a requirement to show these secretquestions, interactive ads as MMOs from the corresponding ad type arealso shown as part of the challenge. The combined challenge is used toserve the purpose of HIP mechanism and also to uniquely identify theuser.

For example, a website requires a set of secret questions to be answeredby a user to uniquely identify the user. As above, such questions may be“what is the model of your first car?” or “what is your pet's name?” or“which city were you born in?” In this example, the questions may becategorized as “automobile”, “pets”, and “travel”, respectively, for thequestions mentioned. Thus, every time these questions are posed, thereis an interactive ad from an automotive industry or a pet industry or atravel industry shown based on the corresponding secret question shown.Solving this interactive ad along with the answer to the secret questionuniquely identifies the user.

An embodiment of this invention is described hereinbelow.

1. A user goes to a website which needs to identify users uniquely. Alsothe website uses HIP to filter out the non-human traffic.

2. The user is required to choose answers for the secret questions askedby the website.

3. The website may have a standard set of secret questions and thesequestions may be categorized under certain ad types.

As in the example above, “What's the model of your first car?” may becategorized under auto, consumer ad categories; and “what's the name ofyour first pet?” may be categorized under pets, family. This mapping ofad categories with the secret questions may be predetermined and storedin the database/storage repository.

4. At the time of creating a user profile as in user signup or userregistration, the secret questions are asked to the user to provide thevalid responses. These questions are presented as a predetermined listreceived from the database. The user provides responses and onsubmission, the questions and responses are stored in the database andare associated to the user id of the user.

5. Next when the user identification is required and also when there isa requirement to filter out no-human traffic (HIP), the user is posedwith the secret question. Along with the secret question, he is alsoposed to interact with the MMO as part of HIP requirement. MMOS checksthe database for the questions for the user id. The questions returnedcause the system to look up the ad categories associated with thequestion. An interactive ad may be fetched for the category and the userprofile data for the user id, question data from the question id, andthe ad data for the ad are provided by MMOS.

6. The webpage displays the interactive ad as a challenge in a HIP form(e.g. type-in, video, and audio) and has a placeholder attribute tocapture the user response. Examples of user responses may include butare not limited to the text entered as a response, the attributes of aclick, the co-ordinates of the drag and drop action on the MMO, theresponse from haptic technology on supporting devices, and touch andtactile sensors on supporting devices. This is in addition to the secretquestion which also needs to be answered.

7. The user interacts with the video and the interaction attributes aresent to the validation server (VS). Examples of interaction attributesmay include but are not limited to the text entered as a response, theattributes of a click, the co-ordinates of the drag and drop action onthe MMO, the haptic technology on supporting devices, and touch andtactile sensors on supporting devices such as but not limited totouch-screen monitors, computers, smart-phones, game consoles.

8. The Validation Server (VS) validates the response attributes to thechallenge served for the MMO and verifies with the dataset for apossible match.

9. When the response satisfies the challenge, a success message is sentto the webpage by the MMOS and statistics about the request is sent tothe AS. Examples of such statistics may include but are not limited tothe timestamp, website which requested, user details if exists, and IPaddress.

8e1. When the response does not fulfill the challenge posed, VS sends afailure message to the webpage and updates the SUS with the failedstatus.

8e2. The webpage indicates failure to the user and sends a request for anew MMO with a challenge to the MMOS, when the threshold for allowablefailures is still unreached.

8e3. The process then repeats at Step 4. Until the number of failuresthreshold is reached by the website or a success is met.

An embodiment can be understood with reference to FIG. 5, which is aschematic diagram of two sample secret questions and their respectivesample MMO with HIP enabled. FIG. 5 illustrates how secret questions maybe used to validate a user's identity and are classified with certain adtypes. Every time the secret questions are shown, corresponding ads arealso shown and used as a challenge. Regarding FIG. 5, the first secretquestion asks the user “What is the name of your car?” 501 and a textbox in which the user can type in the name is provided 502. The useralso is prompted to “click on the car in the image” 503 and an imagewith a car is shown 504. The second secret question is presented andsays, “What is your favorite food?” 505 501 and a text box in which theuser can type in the favorite food is provided 506. The user also isprompted to “type in the name of the restaurant advertised” 507 and acorresponding advertisement is shown 508 along with an input field inwhich the user can type in the name 509.

MULTI-PHASE VALIDATION USING THE CONTENT AS ONE OR MULTIPLE SOURCES ANDTHE INTERACTION AS ANOTHER SOURCE

In an embodiment, interactive multimedia object (MMO) serves hipfeatures and a valid response may be a combination of response from theimage itself and response dependent on the content where such MMO isbeing served. As such, this is a multi-phase validation using thecontent as one or multiple sources and the interaction as anothersource.

In an embodiment, websites use MMO ads for HIP. In this embodiment, thechallenge posed by this type of MMO is split and the response is acombination derived from interacting with the image and also from thecontent being served.

For example, a user may be challenged by a soda MMO ad which asks theuser to type-in the brand name in the ad along with the publishing dateor author of the content where the MMO ad is being served.

An embodiment is described as follows.

A user visits a website which uses MMO ads for HIP.

The webpage calls the multi-media object server (MMOS) to fetch amulti-media object (e.g. an image, video or audio) with additionalattributes sent to MMOS such as website details, user demographicdetails, behavioral pattern of user, etc., when available.

MMOS parses the details sent by the publisher's webpage and fetches arelevant MMO which fits the requirements sent in by the webpage. Thedata to match and serve is stored in the database on the server. MMOScreates a session id for the request.

Along with the image and html content sent to serve the image on theclient, a client side script to be executed on the client such as abrowser or app, is also sent. The client side script may be in any ofthe known client side scripting languages such as javascript, jquery,and actionscript.

The script may parse the webpage's Document Object Model (DOM), which isa cross-platform and language-independent convention for representingand interacting with objects in HTML, XHTML, and XML documents. Objectsin the DOM tree may be addressed and manipulated by using methods on theobjects. The script then extracts the html content of the page. Thecontent may be stored in the web storage of the browser as sessionstorage. For example, Web storage and DOM storage are web applicationsoftware methods and protocols used for storing data in a web browser.

The script may have randomizer logic to pick random words from thecontent of the page and the associated position in the document such asthird word in the first paragraph or first word in the title of thedocument. The word along with the position in the document may be sentto MMOS along with the session id. The script creates part of thechallenge in the form of “tap or type in the third word in the firstparagraph” and displays such along with the MMO rendered on the webpageor app.

Part of the challenge may be interacting with the image/MMO and theother part may be from the content or rest of the webpage which may bedynamic in nature and which may be formed by the script as explained inthe previous step.

The user interacts with the MMO and the interaction attributes are sentto a Validation Server (VS) as response attributes. Examples of suchinteraction attributes include but are not limited to the text enteredas a response, the attributes of a click, the co-ordinates of the dragand drop action on the MMO, the haptic technology on supporting devices,and touch and tactile sensors on supported devices such as but notlimited to touch-screen monitors, computers, smart-phones, and gameconsoles.

The Validation Server (VS) validates the response attributes to thechallenge served for the MMO and verifies with a dataset for a possiblematch. It should be appreciated that this step may be only with respectto the partial challenge which is relevant to the MMO and not with thechallenge from the webpage/content.

When the attributes of the response satisfy the challenge, a successmessage is sent to the webpage and the response pertaining to thewebpage/content is sent back to the publisher.

The publisher may validate the partial response related to thewebpage/content and also check the success/failure from the MMOinteraction sent by VS.

When both responses to both challenges are successful, the validation iscomplete with success as the result.

When the MMO interaction response does not satisfy the challenge posed,VS sends a failure message to the webpage and updates the SUS with thefailed stats. Such failure message may indicate the publisher to ignorethe response related to content/website and consider the wholevalidation as failure. Similarly multiple phases of validation from thewebsite may be used in combination with the MMO interactivity to treatthe whole validation process as a complete unit.

An embodiment can be understood with reference to FIG. 6, a samplewebpage or application demonstrating interactive multimedia (MM))serving HIP features, in which the valid response for this is acombination from the image and response dependent on the content wherethe MMO is being served. Such example shows a multi-phase validationusing the content as one or multiple sources and the interaction asanother source. A webpage or application 601 contains an article aboutpresidential selections 602 and a place where a user may leave a commentabout the article 603. The user is presented with an interactive MMO 604with accompanying instructions, “Click on the wheels of the car and typethe published date of the article” 605. A GUI element is presented forwhich the user to enter the published date 606. It should be appreciatedthat the correct date is Nov. 10, 2000, which a bot would most likelynot be able to detect.

USE OF WEB MAPPING TECHNOLOGY TO ACT AS AN INTERACTIVE MMO THAT HELPSIDENTIFY THE USER AS WELL AS SERVE THE PURPOSE OF HIP

In an embodiment, web mapping software may be used widely by users andthe usage may be becoming more prevalent with smart devices. It shouldbe appreciated that embodiments herein contemplate using any web mappingsoftware as is well known by people with ordinary skills in the art.Such web mapping technology may include the user tagging withinformation useful to the user and may be used as a security mechanismto uniquely identify the user as well as use it for HIP.

For example, a website might ask the user to identify details such as:the city where the user was born; the place where he grew up; thelocation of school; where does he generally stop to fill up gas whencommuting from work to home, point out the location of the favoriterestaurant when he was a kid; the location of the dealer where hepurchased his current/first car, and so on.

An embodiment of this invention is described below.

A user goes to a website which needs to identify users uniquely. Thewebsite uses HIP to filter out the non-human traffic and the website hasgeographical information about the user.

Web mapping software is used to ask the user to identify/tag importantinformation on the map which only the user may be able to identify self.

The website uses this information to help identify the user uniquely.

The validation process may be the same as the validation processexplained in the earlier cases and the process of validation asillustrated in FIG. 1.

An embodiment can be understood with reference to FIG. 7, a schematicdiagram of a webpage or application 701 that uses web mapping technologyto act as an interactive MMO that helps identify the user as well asserve the purpose of HIP. In this example, the user is instructed toclick on the street where he met his wife the first time.

An embodiment can be understood with reference to FIG. 7A, also is aschematic diagram of a registration webpage or application that uses webmapping technology to act as an interactive MMO that helps identify theuser as well as serve the purpose of HIP. Two secret questions are askedof the user and the user responds each time by tapping on the street ininteractive map at the correct location.

For example, a website requires a set of secret questions to be answeredby a user to uniquely identify the user. Examples of such questions mayinclude but are not limited to “Click/tap on the map where you met yourspouse” or “Tap on the street where you attended your kindergarten.” Inthese examples maps are shown at the time of user profile creation whichmay be at the time of user signup or registration. The coordinatesidentified by the user may be stored in the database. Solving thisinteractive ad along with the answer to the secret question uniquelyidentifies the user. An exemplary flow in capturing and associating thedata to a user id is as follows:

Step 1) In a user signup/registration form, user profile details such asUserlD, password, address, etc., are requested. As well a map interfaceis presented to create secret questions and answers. Session informationfrom the page is sent to MMOS to provide a unique id for the user.

Step 2) Maps are shown on the client side along with the predeterminedsecret questions and illustrated in FIG. 7 a. The questions are from thedatabase on the server side and are sent to the client by MMOS, for thatsession id.

Step 3) The user inputs his profile and marks or taps on the map to markthe co-ordinates which tie to the valid response for the secretquestion. The latitude, longitude co-ordinates returned from the map apibe recorded by the client. The map api may be any industry standard apisuch as but not limited to mapquest api or google maps api. Theco-ordinates of the nearest intersection streets from the latitude andlongitude co-ordinates are recorded on the client side. Theseco-ordinates and the user profile details are sent from the clientwebpage to MMOS, along with the session id and user id for that session.Another variation of the map co-ordinates are to obtain the latitude andlongitude of the co-ordinates of the tap/click/address mentioned andthen specify a polygonal area in which the co-ordinates arecircumscribed. The polygonal co-ordinates may be stored in the database.

Step 4) The details are stored in the database and the co-ordinates areassociated to the question and the user id. This information may be usedto validate the user as part of the secret question validation.

Similarly, when needed to be used as a HIP proof, the specifiedco-ordinates may be specifically asked by the secret question as “Tap onthe main street in the map to prove that you are a human and not a bot”.The question and valid answer may be stored in the database and thevalidation of response is as illustrated in FIG. 1.

METHODS TO CREATE A SCALABLE CHALLENGE FOR A MEDIA TO PROVIDE HIPSUPPORT

Multimedia objects (MMOs) can be misused in various ways. Humans or botscan interact with the MMOs to earn revenue and misuse online services.Multimedia objects can be directed to third parties to interact with themedia objects for resolving human proof challenges.

Embodiments herein provide various methods to handle such misuses. Onesignificant issue in designing such methods is to make the challengescalable such that bots find it difficult to predict and recognize.However, to increase media interaction and media engagement, it also maybe necessary to form a relationship of challenges with the mediaobjects. It is desirable that either of these actions be done whilekeeping an easy usability.

In one or more embodiment, various interaction techniques may be used toform such challenges, which may be text based or non-text based.

For text based challenge creation, when metadata of the media isavailable, the techniques used in accordance with embodiments herein usebackend support such as search engines, own devised web crawlers, owndatabase, old media archives, etc., to find relevant information tocreate challenge questions. For example, when the metadata of the mediais “ABC Corp”, the system may find relevant ad slogans of “ABC Corp”from the internet and create a challenge for users to input that slogan.The text may be presented in the form of HIP enabled images to make surethat bots find it difficult to recognize.

For text based challenge creation, when only limited metadata isavailable, the limited data are processed to bring out more sensiblekeywords to be used as meaningful metadata. Embodiments herein processlimited or indirect metadata to a more meaningful metadata. Theextracted information are then fed to the system's backend engine tofind relevant information which may then be used to form challenge sets.For example, the limited metadata for a media may only be a mediaredirect web link. The web link may contain several ids which may notmake any direct sense to filter information. However, the system'sengine makes recommendations and ranks usable words appropriately basedon high hit ratio and other ranking methodologies within temporal andspatial boundaries. One or more embodiments base the challenge questionon the information retrieved. The data for text based challenges may beobtained from any search based api and the procedure may be similar towhat is in the section “Use of web mapping technology to act as aninteractive MMO that helps identify the user as well as serve thepurpose of HIP.” In an embodiment, instead of the map api, search apimay be used.

For text based challenge creation, when no metadata is available, anembodiment may also provide a method to perform image search of themedia to find other similar media and its associated metadata. Thosemetadata are then processed to build challenge questions and are taggedwith the media. This allows close association between media and thechallenge question.

For non-text based challenge creation, when metadata is available, oneor more embodiments may suggest using image object search with themetadata. The metadata based information retrieval system usesintelligent machine learning based system to filter information whichmay then be used to form challenge and response pairs for the media.

For non-text based challenge creation, when no metadata is available,one or more embodiments may suggest to use image search of the media tofind relevant image to be used as a challenge. The response may also beframed using that information.

An embodiment for creating challenges that can scale for a media withHIP support can be understood with reference to FIG. 8, a flow diagramof such process. A browser or web or mobile client (“browser”) sends arequest for an MMO and challenge (HIP) (801). It is determined whetherthe request is for a text based challenge or a non-text based challenge(802). When the request is for a text based challenge, the system getsan MMO suitable for a text based challenge (803). Then the systemdetermines whether the metadata available is sufficient, limited, ornone (804). When the metadata available is sufficient, control goes tochallenge processing data (805). When the metadata available is limited,the system goes to a metadata processing engine (806) and asks whetherit has data to use as a challenge (807) and, if no, goes to a metadatafinder engine (808) and if yes, goes to challenge processing data (805).When the metadata available is none, the system goes to said metadatafinder engine (808). The metadata finder engine determines whether thereis sufficient metadata (809) and when yes goes to processes challengedata (805) and when no, discards the MMO (810). After challengeprocessing data, the system goes to validation processing engine (811)and subsequently to an HIP formation engine (812) and sends said MMOwith said HIP to a client, where the user finds the HIP and interactstherewith (813). After interaction with the HIP at the client, aresponse or request is sent to VS (814) and it is determined whethersaid response is valid (815) and when yes, said client receives apositive response (816) and when no, make a request to said browser toresend request for HIP (817). When said request is for a non-text basedchallenge, the system gets an MMO suitable for a non-text challenge(818) and goes to the step determining whether the metadata available issufficient, limited, or none (804).

METHODS TO CREATE HYBRID CHALLENGES AND RESPONSE SYSTEM TO INCREASE THEEFFICIENCY OF HIP

In an embodiment, for image or multimedia based challenge creation, ahybrid challenge model is provided and may be used to increase theuncertainty in the challenge. In the embodiment, multiple layers ofinteractions may be added independently, each layer acting as achallenge to the user. Multiple types of challenges can be accompaniedtogether and can be processed independently of each other or can becombined with each other as well. For example, a type-in challenge modellayer can be integrated with an additional and independent click basedchallenge model around the type-in. Such embodiment makes the overallsystem very scalable and secure.

It should be appreciated that the system's capability for creating suchchallenges may also be scaled properly because such creations can bedone independently. In an embodiment, multiple servers can be involvedin creating and caching multiple types of challenges and some of suchchallenges may be randomly picked to pose as an overall challenge to anend user. Such systems and type of challenge creation systems can bescaled by increasing the number of servers and running specializedthread or processes on different servers.

An embodiment can be understood with reference to FIG. 9, a flow diagramof a method for creating hybrid challenges and responses. A website orapplication (browser) requests HIP (901) from a traffic analyzer (902),which is configured to send complexity information about the HIP to ahybrid challenge and response generator (903). The hybrid challenge andresponse generator obtains two or more HIPs from any of a click typechallenge and response generator (904), a drag type challenge andresponse generator (905), a type-in text challenge and responsegenerator (906), and other types of generators (907). A combined HIPgenerator combines said two or more HIPs (908) and the combined HIPs areserved to a client. A user interacts with the HIP (909). After thecombined HIPs are interacted with, a request for validation is sent tosaid VS (910). It is determined whether the validation is successful ornot (911). If yes, a success message is sent to said webpage orapplication (901) and when said validation is a failure, a failuremessage is sent to said webpage or application when a refresh thresholdis expired (912), otherwise a request for HIP again is sent to thetraffic analyzer and increment a refresh counter that is used todetermine when the refresh threshold expires (913).

FIG. 9A shows some examples that demonstrate hybrid challenge andresponse based HIP.

METHODS TO MEASURE ENGAGEMENT ON HIP MEDIA

One or more embodiments define and provide different metrics to measureengagement on HIP media. Examples of such metrics are listed below. Itshould be appreciated that such list is exemplary and not exhaustive andthat one skilled in the art may conceive of other metrics and stillbeing within the scope of the invention.

Interaction details. For example, clicks, drags, etc.

Redirection details. For example, any redirected links that were hit.

Interaction validation—success or failure.

Additional interaction—measurement of interaction such as copy ofparticular things in the image, getting coupons, clicking coupons,providing leads, etc.

Cookies enabled metrics—Using cookies to monitor and measure mediaengagement effectiveness.

Time spent—how much time is spent on the HIP media engagement

Statistical data—Collaborative analysis of engagement on a specificmedia from different engagements.

Demographic analysis—how certain demography can engage in one mediadifferently than others.

Interest metric—The system may be configured to provide an interface toshow the user expression and to measure such expressions acrossdemography.

Social metrics—The system may be configured to provide social sharingfor the HIP media and to include social metrics in the engagement.

METHODS TO PROVIDE MULTIPLE KNOWN IMAGES WITH SOME UNKNOWN IMAGES ACTINGAS NOISE

An embodiment provides a component configured to provide noise in HIPwhich can be tied together with valid objects to confuse bots. In theembodiment, the system creates a challenge by including various knownimages such as a book, tiger, plate, and spoon with some unknown images,which can be randomly picked. Such unknown images may be randomly pickedby mixing certain textures of the known images or from any third partysource. Such unknown images may bring enough noise and create morehurdles for bots to detect the correct image from all the presentedimages.

An embodiment can be understood with reference to FIG. 10, a schematicdiagram of noise elements being added to increase the difficulty forbots to identify the valid objects. Known and unknown images may bemixed to create noise in HIP. Such technique adds an extra level ofdifficulty for bots to find valid objects. However, the system keeps itsimple enough for a human. Known and unknown images are determined by anintelligent image selector 1001 recognizing and storing known images inan image repository 1002. Intelligent image selector 1001 executes anoise generator 1003 to generate unknown images having or being noiseand stores the unknown images in a noise repository 1004. A challengeand response generator 1005 combines images from image repository 1002and noise repository 1004 to create the MMO and challenge (HIP)comprising known images and unknown images 1006.

METHODS TO TRANSLATE LANGUAGES USING HIP MEDIA

In an embodiment, a challenge is asked to an end user and the responsemay be a translated version of the challenge or may be some closelyrelated text. A response may be given by providing a translated text.

In an embodiment, the system may be configured to provide an option bywhich the user is not forced to translate and, instead, goes to orchooses another type of response and challenge mechanism.

It should be appreciated that this kind of challenge and response systemalso creates more problems and makes it more difficult for the humanfarm, a group of people who manually solve captchas and are involved inbreaking captchas.

It should be appreciated that an embodiment may provide a utility tosociety by translating languages. In this embodiment, image labeling maybe done and confirmed using translated text. For example, a challengeasked against an image may be to write the three most important thingsnoticed in the picture in a local language. In the embodiment, theresponse and related code may try to translate internally the locallanguages and match with other responses of other languages which arestored in the system. When the there is a match, the response may beconsidered valid.

In other embodiments, writing translated local keywords for an image maybe augmented by other challenges to ensure that the responded answersare valid. As well, more statistical approaches may be taken to confirmthe translated text. For example, a same image can be shown to multiplepeople and when the responses show, via computed statistical analysis, apositive pattern for the valid answer, subsequently, the image may belabeled with keywords, from the multiple responses, with higherprobability of corrections.

METHODS TO PROVIDE SUPPORT FOR IN-PREMISE AND HOUSE-MEDIA TOPUBLISHERS/ADVERTISERS/MEDIA PLATFORMS

In accordance with an embodiment, the system may be configured toprovide flexibility and control to use house-media such as for exampleads for the products owned by the company, ads for the services of thecompany, ads for the products of partners, etc.

Such house-media may be ads or any other branding or promotionalmarketing campaign objects.

In an embodiment, such house-media may be used as HIP media. Challengesand responses may be associated by using any of the techniques mentionedherein. This embodiment enables enterprises such as but not limited tobanks, corporate websites, etc. to brand their own product and serviceswhile providing value added spam protection services.

The system or components thereof may also be housed in the premise ofthe company or any entity that likes to use house-media. This capabilityprovides the company or entity more flexibility and control in using HIPmedia.

ENHANCED AUTHENTICATION AGENT

In an embodiment, the system is configured to provide ways to increasethe secured authentication process. In some of the prior art systems,companies ask users to select an image and some random text for theiraccount. The companies ask users to provide a password only when theysee their selected image and text every time they login. They alertusers not to provide the password if they don't see the image which theyselected for their account. This helps companies to fight phishingissues.

One or more embodiment provides other capabilities to provide an extralevel of authentication protection. For example, the system may beconfigured to ask the user to select some challenges and responses foran image which he or she sees in his or her personal account. In thiscase, users do not just see the image but act on it based on thechallenge asked. In this embodiment, all the challenges and responseinteractions are chosen by the user, such that only the user himself isable to make the correct response. For example, an user can upload animage of their childhood birthday party. The user can create challengessuch as “Who is my best friend?”, “Where is my mom?”, etc. The responsemay be tagged to that image by various interactions such as text input,click, drawing, etc.

In an embodiment, users may upload multiple images and sets ofchallenges and responses to each image to personalize and make theiraccount more secured.

This system helps the companies to validate the user interaction apartfrom the login and password. Because these interactions and challengesare very easy and personalized in nature, these features may increasethe authentication security to a higher level with a very minimal userimpact and cost. With this process not only are companies able to provethemselves as genuine to the users, users may also perform or take extraactions to prove their authenticity.

It should be appreciated that this embodiment of enhanced authenticationagent may be very effective for the small screen and touch devicesbecause interactions are very easy.

In an embodiment, the system is configured such that such requiredinteractions also may be sent to other devices which are capable ofmaking interactions.

METHODS TO PROVIDE FLEXIBILITY TO PICK MEDIA SOURCE

In an embodiment, a component of the system allows flexibility to theuser to pick the media source which can be converted to HIP media by theprocessing done in another component of the system. For example, aparticular image directory path is input to the system which enables thesystem to know from where the images can be picked and used as a HIPmedia.

In an embodiment, such images may be split into pieces, e.g. five,programmatically and stored the correct order into the system as aresponse. A challenge may be created for the user to arrange the splitpicture pieces into an order which is more meaningful and complete. Anyresponse is matched with the stored order and if it matches, the systemvalidates the response.

Similarly with text ads or other texts found in search engines and otherweb sites, in an embodiment the system is configured to use the similartext ads and convert them into HIP. For example, some portion of thetext may be converted into handwritten style by adding some noise. Suchtypes of conversion of the text may be used as a challenge part in HIP.

METHODS TO PROVIDE HIP SUPPORT TO SMART TV

It has been found that current HIP technologies may be very difficult todeal with when HIPs are presented to other smart devices such as a smarttelevision (TV). In an embodiment, the system is configured to sense ordetect and determine various interactions done in a smart TV and tounderstand such interactions as a response. For example, a few patternsof remote control actions may be understood as a response. In anembodiment, the system may be configured to detect if the device is asmart TV and, accordingly, pose a challenge which can be easilyresponded by a TV user.

This embodiment is important to secure the attacks from malicious botssuch as the bots which can fake as a TV device and attack the content orrecord the videos from the TV when it is streamed automatically from webdevices.

TEMPLATE BASED HIP

In an embodiment, a method allows to create several templates havingseveral fixed parts but with some empty spaces. These spaces may be usedfor creation of random challenges. The empty spaces may also berandomized within the template. Such templates may be any multimediasuch as images, animation, or interactive games.

In an embodiment, the templates consist of a structure which is verybasic in nature. Such type of template may be considered a skeleton.Subsequently, different modules, which are part of this template, may bedefined. Also subsequently, a relationship may be built between modulesto propose challenge and response mechanisms. Such template-based HIPprovides a very scalable and uniform way of presenting HIP.

As well, the system is configured to provide capabilities to find andfill the templates automatically given the set of categorized multimediaobjects. The system uses intelligent classification and aggregationalgorithms to populate and build relationships within the templateelements. Based on the template elements, the system may also pick themost suitable relationships to be defined as a challenge in the HIP.

In an embodiment, the response may also be defined using either staticmapping of relationships to the interactions or the response may bepicked dynamically at run time. As well, a challenge may be associatedwith multiple types of responses and vice-versa.

For example, a pattern matching template system generates challengesbased on the pattern matching system. For example, there may be onemultimedia source which has a typical pattern and the challenge imagesmay then be created that match closely with that pattern. However, itshould be appreciated that an embodiment makes it difficult for currentpattern matching algorithms to do reverse matching but make it easy forhumans to solve the challenge.

An embodiment can be understood with reference to FIG. 11, a flowdiagram of creating a template based HIP. An HIP request 1101 is made toa template selection engine 1102. Template selection engine 1102 obtainsa template from a template repository 1103. Template selection engine1102 sends the template to an object relationship selection engine 1104.Object relationship engine uses intelligent classification andaggregation algorithms to populate and build relationships withinelements of said template and based on said elements, picks mostsuitable relationships to be defined as a challenge in the HIP. Based onthat information, an MMO object selection engine 1105 selects an MMOobject. An HIP builder 1106 combines the MMO object and the definedchallenge and sends the combined MMO object with the defined challengeto a client 1107.

An embodiment can be understood with reference to FIG. 12, an examplechallenge to demonstrate the template based HIP and examples ofrelationships, objects, interactions, and templates. FIG. 12 shows anexample of a template based HIP where the template is source and targetcontainers. Objects, relationship between objects, and interaction arealso defined to complete the template based HIP. Templates,relationships, objects, and types of interactions are encompassed as oneset and a different combination of different or the same templates,relationships, objects, and types of interactions are encompassed as adifferent set and two or more different combinations of the sets may beused to create different HIPs.

It should be appreciated that different e-commerce websites are willingto provide a pattern based search of the products. For example, suchwebsites enable a user to take a picture of a product and then searchfor the product in the database of the website. In these types ofdatabases, categorization is a problem. Categorizations andclassifications may be improved using embodiments herein. For example,in an embodiment, the system provides HIP methods that may be employedto ask the user to correctly match patterns. Some of the patterns may beused as a dummy for categorization while others may be used forsecurity. As well, in an embodiment, statistical analysis for some setof population may be used to find which pattern matches closely to theones being asked in the challenge.

It has been found that there are various places where HIP multimediabecomes very important to provide good level of authentication andsecurity. For example, in an embodiment, the system is configured toprovide integration with the tools and techniques that are associatedwith those user flows. For example, downloads require HIP protection.There are several download managers which are out in the market and thesystem herein is capable of being configured to integrate with suchtools.

One good example are comment boxes. Comment boxes are tools by whichusers leave some comment or collaborate on others' comments. Users caninteract with the previous comments such as vote, like, dislike, score.Such interactions are prone to bot attacks. Such tools may be integratedwith the HIP enabled MMOs. In an embodiment, the MMOs also may beattached without HIP enablement in some places to increase theeffectiveness of MMOs.

HUMAN COMPUTATION METHODS USING OUR SYSTEM

In an embodiment, human computations may be used to help systems tolearn and create better training data for the machine learningalgorithms. For example, in an embodiment, a couple of questions may beasked to a user—one for human verification and the other for unsolved orcomplex pattern matching. An embodiment provides such questions suchthat the user may be caused to or is encouraged to try to answer bothcorrectly. Such technique enables the system to learn and optimizeindexing of close matching images.

In an embodiment, the system is configured to help resolve outlierswhich are not being solved by the current computer vision techniques.The system is configured to use human intervention to improve imagepattern searching.

In an embodiment, the system is configured to improve multimediaclassification. For example, the system may provide two multimediaobjects such as images and create a challenge by asking a user tointeract with the common objects in the images. In the example, thesystem may present one image with many objects, e.g. sun, mountain,trees, fruits, and a girl, and a second image that contains a fruit. Thechallenge for the user may be to encircle the object in the first imagewhich belongs to the same class/family/category of the object beingshown in the second image. It has been found that this challenge is adifficult problem for machines to solve. There are recognition,classification, segmentation problems involved in solving the challenge,which makes the challenge difficult for machines to solve easily.

An embodiment provides a system configured to improve multimediaaggregation by providing multiple images and asking users to selectthose that appear more than once. The challenge may also provide hintsto users for selecting images. In an embodiment, the response may be aninteraction on all the images being presented in the challenge. Forexample, the challenge may be to select all images which are owned by asingle company.

In an embodiment, the system is configured to be used in other ways toutilize human computation. For example, the system may be configured torecognize and understand likes or dislikes patterns for a particularproduct under the constraint of a demography, gender, etc., and use theknowledge gained to solve computational problems.

In an embodiment, the system is configured to provide robust and deepanalytics which may be a strong input for several decision makingprocesses and systems.

An embodiment can be understood with reference to FIG. 13, a flowdiagram of creating an HIP system which can integrate some humancomputation methods to utilize human interactions to solve somecomputational problems. That is, the system uses human computationmethods to extract some values out of human interactions apart fromusing HIPs for validation of human interaction. A client 1301 sends aHIP request to a human computation HIP generator 1303, which sendscontrol to an active (real) challenge and response generator 1304 toobtain a part of the challenge which is active and sends control to apassive challenge and response generator 1305 to obtain a part of thechallenge which is passive. The active and passive parts are sent to aHIP builder 1306 which is configured to hide or unhide active andpassive parts of the challenge to create a final challenge. Active HIPvalidation of the active parts of the challenge is performed by a VS1307 and when validation is successful, the HIP response is sent towebsite or application 1301 and when validation is not successful,control returns to human computation HIP generator 1303. Passive HIPvalidation of passive parts of the challenge is sent to a log database1308, which sends the validation on to an analytics server 1309.Analytics server 1309 is configured to use information about saidvalidation in generating reports 1310.

FIG. 13A is a schematic diagram of one example to find image outliers inone image sets. Finding image outliers in image sets is a toughchallenge for computational systems.

AN EXAMPLE MACHINE OVERVIEW

FIG. 14 is a block schematic diagram of a system in the exemplary formof a computer system 1400 within which a set of instructions for causingthe system to perform any one of the foregoing methodologies may beexecuted. In alternative embodiments, the system may comprise a networkrouter, a network switch, a network bridge, personal digital assistant(PDA), a cellular telephone, a Web appliance or any system capable ofexecuting a sequence of instructions that specify actions to be taken bythat system.

The computer system 1400 includes a processor 1402, a main memory 1404and a static memory 1406, which communicate with each other via a bus1408. The computer system 1400 may further include a display unit 1410,for example, a liquid crystal display (LCD) or a cathode ray tube (CRT).The computer system 1400 also includes an alphanumeric input device1412, for example, a keyboard; a cursor control device 1414, forexample, a mouse; a disk drive unit 1416, a signal generation device1418, for example, a speaker, and a network interface device 1428.

The disk drive unit 1416 includes a machine-readable medium 1424 onwhich is stored a set of executable instructions, i.e. software, 1426embodying any one, or all, of the methodologies described herein below.The software 1426 is also shown to reside, completely or at leastpartially, within the main memory 1404 and/or within the processor 1402.The software 1426 may further be transmitted or received over a network1430 by means of a network interface device 1428.

In contrast to the system 1400 discussed above, a different embodimentuses logic circuitry instead of computer-executed instructions toimplement processing entities. Depending upon the particularrequirements of the application in the areas of speed, expense, toolingcosts, and the like, this logic may be implemented by constructing anapplication-specific integrated circuit (ASIC) having thousands of tinyintegrated transistors. Such an ASIC may be implemented with CMOS(complementary metal oxide semiconductor), TTL (transistor-transistorlogic), VLSI (very large systems integration), or another suitableconstruction. Other alternatives include a digital signal processingchip (DSP), discrete circuitry (such as resistors, capacitors, diodes,inductors, and transistors), field programmable gate array (FPGA),programmable logic array (PLA), programmable logic device (PLD), and thelike.

It is to be understood that embodiments may be used as or to supportsoftware programs or software modules executed upon some form ofprocessing core (such as the CPU of a computer) or otherwise implementedor realized upon or within a system or computer readable medium. Amachine-readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine, e.g. acomputer. For example, a machine readable medium includes read-onlymemory (ROM); random access memory (RAM); magnetic disk storage media;optical storage media; flash memory devices; electrical, optical,acoustical or other form of propagated signals, for example, carrierwaves, infrared signals, digital signals, etc.; or any other type ofmedia suitable for storing or transmitting information.

Further, it is to be understood that embodiments may include performingoperations and using storage with cloud computing. For the purposes ofdiscussion herein, cloud computing may mean executing algorithms on anynetwork that is accessible by internet-enabled or network-enableddevices, servers, or clients and that do not require complex hardwareconfigurations, e.g. requiring cables and complex softwareconfigurations, e.g. requiring a consultant to install. For example,embodiments may provide one or more cloud computing components thatenable users, e.g. users on the go, to be connected to, use, create, orimplement scalable and effective multi-media objects with HIPcapabilities on such internet-enabled or other network-enabled devices,servers, or clients. It further should be appreciated that one or morecloud computing embodiments enabling creating and implementing or usingscalable and effective multi-media objects with HIP capabilities usingmobile devices, tablets, and the like, as such devices are becomingstandard consumer devices.

Although the invention is described herein with reference to thepreferred embodiment, one skilled in the art will readily appreciatethat other applications may be substituted for those set forth hereinwithout departing from the spirit and scope of the present invention.Accordingly, the invention should only be limited by the Claims includedbelow.

1. An apparatus for generating and serving scalable and effectivemultimedia objects with human interaction proof (HIP) capabilities on awebpage, comprising: an engagement server configured to receive arequest for a combined multimedia object (MMO) and challenge (HIP), saidrequest originating from a webpage or application, and send saidmultimedia object and said challenge for delivery to the webpage orapplication, wherein the MMO comprises user-generated content insertedinto the media object of the MMO and wherein the challenge comprisesasking a user to interact with a user device in a way that is related tothe user-generated inserted content; wherein the engagement server isfurther configured to receive a response to said challenge originatingfrom the webpage or application; wherein the engagement server isfurther configured to, based on said response, determine a success orfailure by matching said response with a stored response in a challengeand response repository and send a success message or a failure message,respectively, for delivery to said website or application; wherein theengagement server is further configured to receive an acknowledgementwhen said message is said success message and a new request for a newmultimedia object and new challenge until a predetermined threshold ismet, when said message is said failure message; and wherein said MMO andsaid challenge comprise secret questions and answers created by a userand used to validate the user's identity.
 2. The apparatus of claim 1,wherein said secret questions are classified as belonging to particulartypes of advertisements (ads); and wherein when a secret question isshown, a corresponding ad is also shown and used as a challenge.
 3. Acomputer-implemented method for generating and serving scalable andeffective multimedia objects with human interaction proof (HIP)capabilities on a webpage, comprising the steps of: providing anengagement server configured to receive a request for a multimediaobject (MMO) and a challenge (HIP), said request originating from awebpage or application, and send said multimedia object and saidchallenge for delivery to said webpage or application, wherein the MMOcomprises user-generated content inserted into the media object of theMMO and wherein the challenge comprises asking a user to interact with auser device in a way that is related to the user-generated insertedcontent; wherein the engagement server is further configured to receivea response to said challenge originating from the webpage orapplication; wherein the engagement server is further configured to,based on said response, determine a success or failure by matching saidresponse with a stored response in a challenge and response repositoryand send a success message or a failure message, respectively, fordelivery to said website or application; wherein the engagement serveris further configured to receive an acknowledgement when said message issaid success message and a new request for a new multimedia object andnew challenge until a predetermined threshold is met, when said messageis said failure message; and wherein said MMO and said challengecomprise secret questions and answers created by a user and used tovalidate the user's identity.
 4. The method of claim 3, wherein saidsecret questions are classified as belonging to particular types ofadvertisements (ads); and wherein when a secret question is shown, acorresponding ad is also shown and used as a challenge.
 5. The method ofclaim 3, wherein said engagement server is further configured to receivea request for said MMO plus filter criteria comprising profileinformation regarding user identification, further configured to findsaid MMO by performing a matching process with previously stored MMOsusing said filter criteria, and further configured to send said MMO toan MMO server when said matching process is successful; and wherein saidMMO returned to said webpage or application comprises a video that hasbeen previously tagged by having a particular section of the videoassociated to said challenge.
 6. The method of claim 5, wherein saidvideo is previously tagged for secret questions by: uploading a video tobe combined with one or more secret questions; optionally selecting oneor more images to be added to said MMO comprising the video; selectingsaid one or more secret questions to be used when said MMO is served,wherein said one or more secret questions are to be used verify useridentification; selecting one or more frames from said video; for eachof said one or more frames: selecting a section and tagging said sectionand associating said tag with any of said one or more secret questions;and providing an answer for each of said one or more questions;submitting said tagged video to said MMO server and said challenge andresponse repository, wherein said submission comprises any ofinformational data regarding said sections, video, one or morequestions, answers, user identification, password, and user profile; andstoring said submitted data in said MMO server and a challenge andresponse repository.
 7. The method of claim 5, wherein said video ispreviously tagged by: creating a profile; uploading a user specifiedvideo, wherein said user specified video is selected from any of local,public domain, or a private repository; using video editing software,selecting one or more frames from said video; for each of said one ormore frames, selecting a section and tagging said section; storing saidvideo in a video repository and storing metadata of said one or moreframes and associated tags in a metadata repository, wherein said videoand said metadata are recombined when said MMO is served.
 8. The methodof claim 5, further comprising: creating said MMO and said challenge bystitching a user specified video and user defined multiple images andusing the resultant multimedia as a challenge to identify the correctuser, by: receiving a video file; splitting said video file randomlyinto sequences pieces; receiving user defined images; associatingchallenges and responses to the pieces and the images; stitching saidimages with said video pieces and creating a new video file; andassociating said new video file to a profile; wherein said images withinsaid new video file are used in said challenge.
 9. The method of claim3, wherein a valid response to said challenge comprises a combination ofinteraction with an image on said MMO and a response that is dependenton content displayed on said webpage or application.
 10. The method ofclaim 3, further comprising: creating one or more interactive mapscomprising said MMO and said challenge by incorporating web mappingtechnology; wherein said web mapping technology asks a user to identifyor tag important information on said one or more interactive maps whichonly the user may be able to identify.
 11. The method of claim 3,wherein said sending said request for said MMO and said challengefurther comprises: requesting for HIP, wherein it is determined whethersaid request is for a text based challenge or a non-text basedchallenge; when said request is for a text based challenge: get an MMOsuitable for a text based challenge; determine whether the metadataavailable is sufficient, limited, or none; when said metadata availableis sufficient: go to challenge processing data; when said metadataavailable is limited: ask a metadata processing engine whether it hasdata to use as a challenge and, if no, go to a metadata finder engineand if yes, go to challenge processing data; when said metadataavailable is none: go to said metadata finder engine; wherein saidmetadata finder engine determines whether there is sufficient metadataand when yes, processes challenge data and when no, discards the MMO;wherein after challenge processing data, go to a validation processingengine and subsequently to an HIP formation engine and send said MMOwith said HIP to a client; wherein after interaction with the HIP at theclient, send a response to a validation server to determine whether saidresponse is valid and when yes, said client receives a positive responseand when no, make a request to said browser to resend request for HIP;and when said request is for a non-text based challenge, get an MMOsuitable for a non-text challenge and go to said step, determine whetherthe metadata available is sufficient, limited, or none.
 12. The methodof claim 3, wherein said MMO and said challenge comprise hybridchallenges and wherein said hybrid challenges are created by: saidwebsite or application requesting HIP; said request being received at atraffic analyzer configured to send complexity information about the HIPto a hybrid challenge and response generator; wherein said hybridchallenge and response generator obtains two or more HIPs from any of aclick type challenge and response generator, a drag type challenge andresponse generator, and a type-in text challenge and response generator;wherein a combined HIP generator combines said two or more HIPs; whereinsaid combined HIPs are served to a client; wherein after the combinedHIPs are interacted with, a request for validation is sent to avalidation server; wherein when said validation is successful, saidsuccess message is sent to said webpage or application and when saidvalidation is a failure, send said failure message to said webpage orapplication when a refresh threshold is expired, otherwise request HIPagain at said traffic analyzer and increment a refresh counter that isused to determine when the refresh threshold expires.
 13. The method ofclaim 3, wherein said MMO and said challenge comprise HIP; wherein saidMMO and challenge and HIP comprise known images and unknown images;wherein the unknown images operate as noise in said HIP and cause moredifficulty for bots to find valid objects in said MMO; and wherein saidknown and unknown images are determined by: an intelligent imageselector recognizing and storing known images in an image repository,said intelligent image selector executing a noise generator to generateunknown images having or being noise; said intelligent image selectorstoring said unknown images in a noise repository; said engagementserver comprising a challenge and response generator which combinesimages from said image repository and said noise repository to createsaid MMO and challenge and HIP comprising known images and unknownimages.
 14. The apparatus of claim 3, wherein an HIP is created usingtemplate based HIP building comprising: providing a template selectionengine for receiving an HIP request from the webpage or application;said template selection engine obtaining a template from a templaterepository; said template selection engine sending said template to anobject relationship selection engine; said object relationship engineusing intelligent classification and aggregation algorithms to populateand build relationships within elements of said template and based onsaid elements, picking most suitable relationships to be defined as achallenge in the HIP; an MMO object selection engine selecting an MMOobject; an HIP builder combining the MMO object and said definedchallenge and sending said combined MMO object with said definedchallenge to a client.
 15. The method of claim 14, wherein templates,relationships, objects, and types of interactions are encompassed as oneset and a different combination of different or the same templates,relationships, objects, and types of interactions are encompassed as adifferent set and wherein two or more different combinations of saidsets are used to create different HIPs.
 16. The method of claim 3,wherein said request for said MMO and said challenge comprises an HIPrequest and wherein sending said request for said MMO and said challengefurther comprises: receiving by a human computation HIP generator, theHIP request, which sends control to an active (real) challenge andresponse generator to obtain a part of the challenge which is active andsends control to a passive challenge and response generator to obtain apart of the challenge which is passive; sending said active and passiveparts to an HIP builder which is configured to hide or unhide active andpassive parts of the challenge to create a final challenge; whereinactive HIP validation of said active parts of said challenge isperformed by a validation server and when validation is successful, theHIP response is sent to said website or application and when validationis not successful, control returns to said human computation HIPgenerator; wherein passive HIP validation of said passive parts of saidchallenge is sent to a log database, which sends said validation on tosaid analytics server and wherein said analytics server is configured touse information about said validation in generating reports.
 17. Themethod of claim 3, wherein said request comprises information about adevice from which said website or application are communicativelyconnected and said MMO server is further configured to parse and usesaid device information to choose said MMO and said challenge based oncapabilities of said device.
 18. The method of claim 3, wherein saidchallenge is challenge asked against an image for a user to provideimportant information noticed in the picture in a local language andwherein related code translates internally the local language andmatches with other responses of other languages which are stored in thesystem and when there is a match, the response is considered valid. 19.The method of claim 3, wherein said MMO comprises in-premise orhouse-media, wherein in-premise or house-media comprise ads or any otherbranding or promotional marketing campaign objects owned by a companyassociated with said website or application or partners of said company.20. The method of claim 3, wherein the engagement server is furtherconfigured to detect if the request comes from a device that is a smartTV and to detect and determine interactions done in the smart TV and tounderstand such interactions as a response.
 21. The method of claim 3,wherein said MMO is configured to be turned on and off for saidchallenge in a workflow.